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Accountability * Integrity * Reliability 



United States General Accounting Office 
Washington, D.C. 20548 



September 10, 2002 

The Honorable Judd Gregg 
United States Senate 

Dear Senator Gregg: 

Privately funded voucher programs are a relatively new development in the 
nation’s experiment with school vouchers. These programs, started in the 
early 1990s, provide low-income families with private, nongovernmental 
tuition assistance at private schools for kindergarten through grade 12. 
While private schools have long offered various forms of financial 
assistance, many of these privately funded voucher programs are different 
from traditional scholarship efforts in two key respects: they are open to 
any applicant solely on the basis of family income level, and recipients are 
free to decide which schools their children should attend. Such programs 
now serve about 46,000 of the estimated 63 million school age children 
nationwide, awarding nearly $60 million in tuition assistance. 

Privately funded voucher programs are becoming part of an evolving 
approach to achieving greater school choice. The Congress continues to 
show a strong interest in school choice issues — debating a number of 
choice issues, enacting choice provisions in Title I of the No Child Left 
Behind Act of 2001, and enacting the Public Charter Schools program and 
the Magnet Schools Assistance program. It was within this context that you 
asked us to provide information on both publicly funded and privately 
funded voucher programs. We addressed publicly funded voucher 
programs in a prior report. 1 This report on privately funded voucher 
programs focuses on answers to the following questions: 

• What are the characteristics of privately funded school voucher 
programs, including such factors as amount of tuition assistance, 
determination of student eligibility, and long-term challenges? 

• What is known about the academic performance of students 
participating in these programs and the degree of parental satisfaction 
with the programs? 



‘U.S. General Accounting Office, School. Vouchers: Publicly Funded Pmgrams in Cleveland 
and Milwaukee, GAO-Ol-914 (Washington, D.C.: Aug. 31, 2001). 
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To report on the characteristics of the programs and long-term challenges, 
we focused on the 78 privately funded voucher programs that were open to 
all low-income applicants and allowed recipients a wide choice in deciding 
what schools their children would attend. We developed our list in 
consultation with officials from two major national organizations with 
which many programs are affiliated and summarized information on these 
programs collected by the organizations. In addition, we searched the 
literature, reviewed reports and other documents, and corroborated 
information through on-site and telephone interviews with local programs 
and national organization officials. To determine what is known about 
student academic performance and parent satisfaction associated with 
privately funded school voucher programs, we relied on existing studies. 
We identified 1 1 studies, but for our review only included those that: 

(1) analyzed student academic achievement and/or parental satisfaction, 

(2) compared voucher students or their parents with an appropriate control 
or comparison group of students or parents, and (3) gathered data on 
student achievement or parental satisfaction both before and after the 
vouchers were awarded. For our analysis, we included findings from those 
studies that reached the 95 percent confidence level as statistically 
significant effects. (There is a 95 percent certainty that these results would 
not occur by chance alone.) See appendix I for detailed information about 
these studies. Two social scientists examined each study to assess the 
adequacy of the samples and measures employed, the reasonableness and 
rigor of the statistical techniques used to analyze them, and the validity of 
the results and conclusions that were drawn from the analyses. We 
conducted our work between October 2001 and April 2002 in accordance 
with generally accepted government auditing standards. 


Results in Brief 


The 78 privately funded voucher programs we reviewed shared numerous 
characteristics and faced common challenges, but the programs varied 
widely in the dollar amount of the vouchers they awarded and the number 
of students receiving them. While all programs used income-based 
eligibility criteria, many used a sliding scale based on family income and 
number of family members to determine award amounts. Programs 
typically required a minimum parental contribution toward the tuition, and 
many automatically accepted siblings and awarded vouchers using a 
lottery. The average voucher amount in school year 2001-02 ranged widely 
among programs from about $600 to about $2,000 per student and program 
size ranged from as few as 4 students, to over 3,000 students. Beyond the 
information needed to determine eligibility and conduct financial 
oversight, most programs reported collecting little data about participating 
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students, their families, or the schools they attended. Looking to the future, 
program officials said voucher programs face two major challenges— 
specifically, sustaining programs for the long term and retaining students in 
the programs. Some programs we contacted have begun developing ways 
to address these challenges. 

Rigorous evaluations of privately funded voucher programs in New York, 
Dayton, and Washington, D.C., provide some evidence that African 
American students who used vouchers to attend private schools showed 
greater improvements in math and reading than students in the comparison 
group, and have also found that the parents of voucher users of all racial 
and ethnic groups were consistently more satisfied with their children’s 
education than parents of comparison group students. More specifically, 
the New York study found consistently greater improvement in math and 
reading achievement for African American elementary students using 
privately funded vouchers. Voucher users in Dayton showed no significant 
improvements in reading or math test scores. The Washington, D.C., study 
demonstrated positive effects for African American students in the second 
year of the study, but these disappeared in the third and final year of the 
study. No significant differences were found in any of the studies for 
students in other ethnic or racial categories. The three studies also found 
that families that used vouchers were generally more satisfied with their 
children’s schooling on such factors as safety and academic quality than 
were similar families in the comparison groups. In addition, parents of 
children using privately funded vouchers reported that their children s 
schools communicated with them more frequently and had a more positive 
environment as shown by less disorderly behavior among students. While 
the results of these studies suggest positive academic achievement effects 
for African American students and families in New York City, the programs 
examined were relatively small in scale, therefore, the findings cannot be 
generalized beyond the specific programs and geographic areas where they 
were conducted. 



Background 



Privately funded school voucher programs got their start in 1991 when an 
Indianapolis businessman founded a local program that provided tuition 
assistance to about 750 low-income students in grades kindergarten 
through 8 (K-8). After the initiation of the Indianapolis program, a number 
of other communities also established privately funded voucher programs 
based on the Indianapolis model and used funding from local donors. In 
1994, a local voucher program that had been founded in San Antonio 
obtained a grant to establish a national clearinghouse organization — now 
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called Children First America (CFA) 2 — to assist new and existing local 
programs with funding and technical assistance. By 1997, there were 
31 local programs offering over 12,000 privately funded vouchers to K-12 
students in 18 states plus the District of Columbia. 

In 1998, a second national organization, the Children’s Scholarship Fund 
(CSF), was established. CSF used the word scholarship in its name — rather 
than voucher— to distinguish itself from publicly funded voucher 
initiatives. CSF helped found several new local voucher programs and 
established partnerships with many existing programs. In its first year, CSF 
provided one-to-one matching funds for a total of 40 programs to provide 
vouchers for low-income students for a period of 4 years. Beginning in 
school year 1999-2000, CSF-funded programs provided school vouchers to 
about 40,000 K-8 students. CSF currently provides support to 47 affiliated 
programs. Nine of the programs are administered by the CSF national 
office, including a program that provides vouchers to students not residing 
in an area with a local program. Like CSF, CFA continues to be a source of 
funding support and technical assistance to local privately funded voucher 
programs. The two organizations work with each other in a number of 
efforts but differ somewhat in their stated missions. CSF offers tuition 
assistance to needy families in what it describes as a purely charitable 
effort, while CFA states that its mission is to promote parental choice, 
including both privately and publicly funded options. 



In addition to programs offering vouchers for low-income students to use, 
with virtually no restriction as to their choice of private schools, there are 
other privately funded tuition support or scholarship programs. However, 
these programs fall outside the scope of our study because they limit 
student eligibility or school choice in different ways. For example, we did 
not review traditional scholarships offered by private schools, merit-based 
programs, or programs that limit choice to schools of a particular religious 
denomination. 

According to a recent survey done by CSF, 78 privately funded school 
voucher programs used family income as their only eligibility criteria and 
permitted families to use their award at nearly any private school. Although 
these programs receive their funding from private individuals and groups— 



Children First America was originally founded in 1992 as Children’s Education Opportunity 
Foundation America (CEO America). 
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Programs Shared 
Characteristics and 
Challenges, but Varied 
in Size and Voucher 
Amounts 



o 

ERIC 



in contrast with publicly funded school voucher programs 3 — they 
nonetheless may affect public funding in several ways. Most programs are 
not-for-profit organizations and, as such, are eligible for the associated 
federal tax benefits . 4 Programs may also have an impact on local public 
school funding because much state and some federal funding to school 
districts is allocated on the basis of formulas incorporating the number of 
students attending the schools. Some states have laws in place to provide 
state tax credits to individuals or businesses for their contributions to 
tuition assistance organizations, including privately funded voucher 
programs . 5 



The privately funded voucher programs we reviewed had a number of 
common characteristics, including several long-term challenges, but they 
varied widely in the dollar amounts of the vouchers they awarded and 
number of students served. Most programs had local financial support. 
Programs also used similar methods to determine student eligibility and 
many programs used a sliding scale based on family size and income. 
Nearly all programs required parents to contribute at least a specified 
minimum amount toward the cost of tuition. Many automatically accepted 
siblings and some ensured multiyear funding as part of an emphasis on 
helping families. Most programs collected only the information they 
needed to determine student eligibility and administer the programs, such 
as family income and number of siblings. However, information on other 
student and family characteristics was limited. Looking to the future, 
program officials said voucher programs face two major challenges— 
specifically, sustaining programs for the long term and retaining students in 
the programs. Some programs we contacted have begun developing ways 
to address these challenges. 



3 See GAO-Ol-914, August 31, 2001. 

‘Under section 501 of the Internal Revenue Service Code, nonprofit organizations are 
exempted from paying federal income taxes. In addition, donations made to these 
organizations are tax deductible. 

^Arizona provides tax credits to individuals for contributions to organizations that provide 
scholarships to students to help meet the cost of private school attendance. Florida offers 
tax credits to coiporations that fund organizations providing scholarships to 
nongovernmental schools. Pennsylvania grants businesses tax credits for contributions to 
organizations that award scholarships allowing children to attend the school of their choice. 
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Programs Had Many 
Common Characteristics 
but Differed Widely in 
Numbers of Students Served 
and Voucher Amounts 



Although scattered across the United States, the local privately funded 
voucher programs we studied shared many things in common. For 
example, most were largely supported by local donations, and the majority 
of the 47 programs affiliated with CSF raised funds locally to receive 
matching funds from that national organization. Often programs were 
supported by one, or a few, “anchor” donors — people or organizations that 
had committed to making a substantial, and often multiyear, financial 
contribution to support vouchers in their community. For example, a single 
local donor committed to provide the matching funds for the Children’s 
Scholarship Fund of Seattle/Tacoma for 4 years, and in San Antonio two 
local donors have provided nearly all of the funding for two voucher 
programs serving that area. 



Programs also had similar voucher recipient selection processes. Most 
programs selected recipients by lottery from among all eligible applicants, 
although a few selected them on a first-come-first-serve basis. According to 
officials, some programs awarded vouchers to all eligible students in the 
first year because they had more money for vouchers than they had 
qualified applicants. As more people became aware of the vouchers and 
demand grew, nearly every program adopted a lottery system. Many 
programs have established separate lotteries for private and public school 
students. Several program officials pointed out that awarding tuition 
assistance to students already in private school was not accomplishing the 
mission of extending school choice because those families had already 
exercised choice. Programs we contacted typically awarded a larger 
percent of their vouchers to students transferring from public schools and 
established a specific limit to the percent that would be awarded to 
students already in private school. For example, the Washington 
Scholarship Fund, in Washington, D.C., reported that it awards at least 
75 percent of its vouchers to students coming from public schools. The 
program maintains two pools of applicants — one for public school students 
and one for private school students — and selects students by separate 
lottery from each pool. 



Eligibility criteria were also similar among programs. Eligibility depended 
on both family income and size. Almost all the programs targeted the 
vouchers to families eligible for the federal free and reduced price lunch 
program those making less than 185 percent of the federal poverty 
guidelines. However, many programs included families with incomes up to 
270 percent of the poverty guidelines. Many programs also used a sliding 
scale based on family income and size to determine both eligibility and how 
large a voucher the family could receive. Families with higher incomes 
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received a smaller percentage of school tuition. The CSF national 
organization has established sliding scale income eligibility standards for 
its affiliated programs. See table 1 for the CSF 2001-02 income eligibility 
scale. Programs also typically had a maximum voucher amount that could 
be awarded. Maximum award amounts for CSF programs varied from 
program to program but ranged from $1,000 to $2,100 for K-8 vouchers and 
$1,000 to $2,900 for high school vouchers. Since voucher amounts did not 
typically cover the full amount of tuition, the family was responsible for the 
balance. 



Table 1: CSF 




Annual Income Eligibility Scale for School Year 2001-02 Vouchers 


Household 

size 


Household income 
level for voucher of 
up to 75% of tuition 


Household income 
level for voucher of 
up to 50% of tuition 


Household income 
level for voucher of 
up to 25% of tuition 


2 


$0- $11,250 


$11,251 -$20,813 


$20,81 4 -$30,375 


3 


' $0- $14,150 


$14,151 -$26,178 


$26,179- $38,205 


4 


$0- $17,050 


$17,051 -$31,543 


$31 ,544 - $46,035 


5 


$0- $19,950 


$19,951 -$36,908 


$36,909 - $53,865 


For oach 


Add $2,900 to 


Add $5,365 to 


Add $7,830 to 


additional child household income 


household income 


household income 



Source: Children’s Scholarship Fund Program Manual. 




Despite the size of a voucher for which a family is eligible, most programs 
required families to contribute a minimum amount toward their child’s 
tuition payment— at least about $500. However, many programs do not 
require families with more than one child receiving a voucher to pay the 
minimum amount for each child. For example, in Buffalo, New York, the 
program required each family to pay just $500 toward the tuition, 
regardless of the number of children receiving a scholarship. 

For many programs, including all CSF affiliated programs, siblings of 
award recipients were automatically offered vouchers. In addition, the 
programs typically made a commitment to the families to provide the 
vouchers for multiple years, as long as the family remained eligible for the 
program. For example, in Cincinnati, all children in the family can receive a 
voucher, and eligible students will continue to receive a voucher for at least 
4 years. Officials said accepting siblings and committing to a multiyear 
voucher were part of their approach to helping the children and the 
families. In addition, CSF plans to continue its commitment to families by 
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extending financial support for vouchers through the year 2013 to include 
all younger siblings through the eighth grade. 

Programs we contacted also conducted similar financial, oversight 
activities. They typically paid voucher amounts directly to the private 
schools in several tuition installments throughout the school year. Prior to 
each payment, programs received confirmation from participating private 
schools that the voucher students were still enrolled. Many programs also 
required confirmation that the students were attending class regularly and 
some required parents to sign a confirmation form each time a payment 
was made. For example, the Houston Children’s Educational Opportunity 
Foundation pays voucher installments to participating schools every month 
during the school year. Prior to each payment, the program official sends a 
commitment form to each school. A school official and the parents of each 
voucher student are required to sign it. The CSF national organization 
directs its affiliated programs to have the parents of voucher students visit 
their schools to sign a Scholarship Verification Report. The form is also 
signed by the school principal and is returned to the program before 
payment is made. Program officials are instructed to retain the forms as an 
audit trail of the scholarships and proof that they were verified. 

Although the programs shared many similar characteristics, they differed 
in the average dollar amount of their voucher awards. Award amounts are 
based, in part, on school tuition and the tuition at participating private 
schools varied considerably among program locations. For example, for 
K-8 voucher recipients in school year 2001-02, the average private school 
tuition in Buffalo, New York, was about $1,500 and the average K-8 voucher 
award amount for the Buffalo program was $620. The maximum Buffalo 
program award was $1,000, depending on family size and income. In 
contrast, the private schools attended by K-8 voucher students in the 
Atlanta CSF program charged an average tuition of about $4,300 and the 
average K-8 voucher amount for the Atlanta program was $1,663 — with an 
award maximum of $2,100. The average percent of tuition covered by CSF 
program vouchers ranged from about 20 percent in Pittsburgh to about 
65 percent in Baton Rouge. Overall, the average K-8 voucher award for the 
47 CSF affiliated programs was about $1,100, according to CSF officials, 
and the average per-child K-8 tuition among all programs that receive 
funding from the CSF was about $2,550. Average high school voucher 
awards from individual programs also varied widely from $1,009 to $2,411. 
The Department of Education reports that the average Catholic or other 
religious private school tuition for grades K-12 in school year 2000 was 
about $2,800 and for private nonsectarian schools was about $8,900. 
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The number of vouchers awarded per program also varied considerably. 
For example, programs in Phoenix; St. Louis; Erie, Pennsylvania; and 
Midland, Texas, awarded 10 or fewer scholarships each, while the largest 
the CSF local program in New York City — awarded over 3,100. Seventeen 
of the local programs we studied awarded more than 1,000 vouchers. In 
addition, the national CSF organization awarded about 3,600 vouchers, 
through its national program, to eligible students not living in a local area 
served by a voucher program. 

Information on the students or families who receive privately funded 
vouchers is limited, as is information on the private schools the students 
attend. Most programs we contacted reported that they collected only the 
student and family data needed to determine student eligibility, and the 
administrative data needed to manage and oversee program activities. The 
eligibility data collected by programs included family income, size, and 
place of residence, and the administrative and oversight data included the 
amounts of the vouchers awarded and the tuition at the schools voucher 
students attended. Information on the private schools students attended 
was often limited to the school affiliation— for example, Catholic, 
nondenominational Christian, or independent. Programs reported that the 
majority of the schools voucher students attend are religiously affiliated. 
Most programs we contacted did not collect information about the public 
schools voucher students previously attended, and most did not gather 
data to evaluate program outcomes or results. While some program 
officials said they saw value in such analyses, several said that evaluations 
were costly and that they preferred to use their limited funds for student 
vouchers. 



Some Programs Initiated 
Activities to Meet Future 
Challenges 



Sustaining Programs for the 
Long Term 



Through our discussions with program officials, we identified two major 
challenges faced by privately funded voucher programs — ensuring program 
sustainability and maximizing student retention rates. We found that many 
of the programs had recognized these challenges and identified or 
implemented activities and initiatives to address them. 



Most programs we contacted were concerned with sustaining their 
activities over the long term. Officials generally said they expected their 



6 Overall, about 78 percent of private schools nationally are religiously affiliated, according 
to the Department of Education. 
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programs to continue operating for the foreseeable future or indefinitely. 
Some had determined that certain program modifications would be needed 
to foster sustainability. Specifically, these programs recognized that they 
should broaden their funding base and have more community 
representation on their boards of directors. Several programs had been 
founded and initially funded largely by a few local donors. In many cases, 
the donors had agreed to provide funding for 4 years. One program official 
said he believed there was a danger of what he called “donor fatigue” and 
determined it could be averted by expanding the number of donors — 
particularly small contributors. Some programs also saw the benefits of 
broadening community representation on their boards of directors. For 
example, the Washington Scholarship Fund told us it is currently building a 
broad donor base and is expanding its board of directors— which had been 
limited to large donors — to include members who may not be able to 
provide funding, but are actively involved in the community. Additionally, 
an official at Children First Utah told us that to help ensure stability and 
sustainability, the program has established a board of directors with broad 
community representation, including representatives from several large 
corporations and a public relations firm, community activists, and a 
representative from the Hispanic community. In addition, Children First 
Utah has implemented efforts to broaden its funding base, including direct 
mail solicitations, media public relations efforts, and several funding 
proposals to foundations. 

The two national private voucher organizations, CFA and CSF, support 
local programs in both implementing and sustaining their activities. In 
addition to providing matching funds to some programs, they also provide 
technical assistance. CFA offers a detailed “how to” manual to new 
programs and will work on site with programs as needed at no cost. CFA 
officials told us they also work with existing programs to help them solve 
problems and enhance program stability, and they emphasized that their 
focus is on the sustainability of the programs. For example, CFA assists 
programs in broadening the community representation on their boards and 
among their donors. According to CSF officials, that organization provides 
its affiliates with extensive guidance and technical support for program 
administration activities such as scholarship disbursement procedures, 
enrollment verification, and financial planning. CSF also holds an annual 
conference for its affiliates, provides a detailed program management 
manual, and assists affiliates in the use of a sophisticated program 
administration database. 
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Maximizing Student Retention 



Recent Research on 
Privately Funded 
Voucher Programs Has 
Explored Academic 
Achievement and 
Parental Perceptions of 
Their Children’s 
Schools 



Maximizing the retention rate among voucher recipients was another 
challenge faced by the programs we contacted. In many programs, about 
20 percent or more of the voucher recipients leave the program each year. 
However, according to a CSF official, only a small percent of voucher 
recipients leave CSF affiliated programs because they graduate from the 
eighth grade. Program officials said that other reasons recipients leave 
might include moving or dissatisfaction with the school. Many said they 
believe family financial difficulties are a common cause of attrition. 
According to some officials, families are sometimes unprepared for all the 
costs of having children in private schools— for example, uniforms, books, 
and activity fees. Additionally, parents of voucher students may be 
confronted with financial emergencies that preclude their paying their 
required portion of the tuition and continuing in the program. Some 
programs we contacted attempted to determine the reasons students drop 
out and several had initiated activities to promote student retention. For 
example, the CSF program in New York City and the Washington 
Scholarship Fund have implemented “stay in school funds” to help parents 
who are experiencing financial emergencies such as unexpected medical 
expenses. Washington Scholarship Fund, which served nearly 900 families 
in school year 2001-02, reported that it has provided such funding 
assistance to about 26 families so far. The Horizon program in San Antonio, 
Texas, initiated a voucher parent group to increase the student retention 
rate. According to a program official, the group, “Las Comadres, meets 
weekly to discuss aspects of their children’s education and learn from each 
other about schools and curricula. The meetings also include guest 
speakers on such topics as preparing children for high school and college, 
parental rights and duties, and children’s health issues. The Horizon 
program director and staff also attend the meetings to facilitate and answer 
questions. 



We identified three studies of privately funded voucher programs— in New 
York; Dayton, Ohio; and Washington, D.C.— that were rigorous enough to 
meet our criteria for inclusion. These studies all included quantitative 
analysis of program effects, used appropriate comparison groups, and 
gathered data before vouchers were awarded, as well as after, to assess the 
equivalence of study groups and to track program effects. In general, 
researchers found that African American students in New York City who 
used vouchers to attend private schools exhibited more substantial 
improvements on test scores in math and reading than African American 
students in the comparison group. For students from other ethnic groups, 
there were no sizable or significant differences in the test score gains of 
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voucher users and comparison group students, or there were too few 
participants to draw any conclusions. These studies have also found that 
the parents of voucher users in all programs were consistently more 
satisfied with their children’s education, regardless of ethnic group. 
Although these results suggest positive achievement effects for some 
African American students using vouchers, they cannot be generalized 
beyond the specific programs, schools, and geographic areas where the 
studies were conducted or the low-income group of families studied. 



Studies Used Same The studies of privately funded voucher programs in New York; Dayton, 

Rigorous Methodology Ohio; and Washington, D.C., all used impact evaluations to study the effect 

of vouchers on private school attendance. Impact evaluations isolate a 
program effect from the effects of other factors that could influence 
participants outcomes. To isolate the program’s effect, impact evaluations 
divide participants into two groups: those who receive program services 
and a similar group who do not (the control or comparison group 7 ). Some 
impact evaluations assign participants randomly to one group or the other. 
Random assignment increases the likelihood that the two groups are 
roughly equivalent on all characteristics that could affect outcomes. It 
helps ensure, for example, roughly equal numbers of very low-income 
students in both groups, rather than those students being concentrated 
more heavily in one group or the other. The use of random assignment to 
create equivalent groups allows researchers to compare tests scores (or 
other outcomes) for the two groups and attribute any differences to the 
program services rather than other factors, such as differences in family 
income. 



The studies of voucher programs in New York, Dayton, and Washington 
randomly assigned applicant families to the voucher offer group and the 
control (no voucher offer) group. Before each voucher program began, 
students whose families hoped to receive a voucher took the Iowa Test of 
Basic Skills in math and reading, and their parents completed surveys 
about family background characteristics and their children’s educational 
experiences. The researchers also screened families to determine if they 
were eligible to participate on the basis of their incomes. Test scores and 
information about family background provided researchers with a baseline 
for making later comparisons. Families who met the income qualifications 



7 When participants are assigned to the group that receives services and the comparison 
groups randomly, as in these studies, the comparison group is called a control group. 
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and participated in the testing and survey sessions were eligible to enter a 
lottery to decide which families would receive voucher offers and which 
would not. Because the researchers ensured that the lotteries were 
completely random, the two groups — voucher offer recipients and 
nonrecipients— should have roughly equivalent average student test scores 
and family backgrounds at the outset. To collect follow-up data from both 
voucher offer recipients and nonrecipients in subsequent years of the 
studies, the researchers used the same testing and survey procedures. 8 
Only students attending a public school before the lottery were included in 
the follow-up testing and analysis. Table 2 provides additional details about 
the studies, including the numbers of applicants and vouchers awarded. 
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According to the researchers, approximately equal percentages of voucher recipients and 
nonrecipients returned for follow-up testing and surveys. 
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New York 


Dayton 


Washington 


First school year under study 


1997-98 


1998-99 


1998-99 


Years of study follow-up 


3 


2 


3 a 


Entry grades of students participating 
in studies 6 


1-5 


1-9 


1-9 


Number of students tested before the 
lottery 


1,960 


803 


1,582 


Number of vouchers awarded 


1,500 


515 


811 


Maximum voucher amount 


$1,400 


$1,200 


$1 ,700 


Number of private schools attended 
by voucher users 0 


216 


42 


116 



Note: The numbers reported here include only students whom the researchers included in the studies 
The programs awarded additional vouchers to students not included in the studies because of their 
entry grade or they were already attending a private school. 



Although the Washington study lasted for 3 years, complete results from the third year follow-up 
sessions were unavailable at the time this was written. Only composite test scores (combined readina 
and math) were available. Individual reading and math scores and the results of the parental survey 
had not yet been released. 

Students entering first grade were not tested at baseline, and students did not take follow-up tests 
beyond eighth grade. K 

oT!in S n^ nUml3erS re P resent the number of schools attended by all of the programs’ students in the 
2000-01 school year, not just by those included in the studies. Because not all students in the 
programs participated in the studies, the number of schools attended by students participating in the 
stuaies is iikeiy to be somewhat smaller. K y 



Source: William G. Howell and Paul E. Peterson, The 
(Washington, D.C.: Brookings Institution Press, 2002). 



Education Gap: Vouchers and Urban Schools 



While the use of control groups helps isolate the cause of any outcome, 
researchers must still address circumstances that threaten the quality of 
their results. In the New York, Dayton, and Washington studies, the 
researchers addressed two such circumstances. First, not all of those 
offered a voucher used their voucher to attend a private school, and some 
students who did not receive a voucher offer ended up in a private school 
anyway. This meant that a comparison between the voucher offer group 
and control group, as designed, revealed the effect of offering a voucher to 
eligible students in a city, not the effect of actually using it to attend a 
private school. To answer the question of the effects of vouchers on those 
actually using them, the researchers used a data analysis technique that 
allowed them to compare the outcomes of those who actually used a 
voucher to attend private school to those of students who were most likely 
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to have attended private school had they been offered a voucher. 9 While 
both the voucher offer and voucher use analyses are valid, they answer 
different questions. The researchers reported primarily on the voucher use 
analysis. Second, substantial numbers of those who initially agreed to 
participate in the studies dropped out in subsequent years. When this 
occurs, those who remain in the study may be different from those who 
left, which can threaten the equivalence of the two study groups, making it 
difficult to assess the effect of the voucher offer. The researchers used a 
standard statistical procedure 10 to minimize the possibility that the loss of 
some participants undermined their results. 

In all three studies, the demographic characteristics of study participants 
surveyed at baseline reflected the programs’ targeting of vouchers to low- 
income families, most of whom lived in inner cities. Although these families 
met program eligibility requirements as low-income, they were able to pay 
at least a portion of the private school tuition. In both Dayton and 
Washington, the average income of a participant family that received a 
voucher and used it to attend private school was approximately $18,000. 
However, in New York, the average family income of participating voucher 
users was about $10,000, possibly reflecting the more stringent income 
criteria of the New York program. In all three cities, ethnic minorities made 
up a large majority of voucher applicants participating in the studies. 
According to the survey questions, study participants were asked to report 
their race as either Black/African American (non-Hispanic), White (non- 
Hispanic), Puerto Rican, Dominican, other Hispanic, American Indian or 
Alaskan Native, Chinese, other Asian or Pacific Islander, or other. Thus, the 
African Americans and Whites in the study are likely to be non-Hispanics 
only. (See table 3.) Among study participants who made use of the voucher 
offered to them, a majority attended Catholic schools in all three cities. 



°The method employed was an instrumental variable technique. For additional information 
on this and other aspects of the studies’ methodology, see William G. Howell and Paul E. 
Peterson, The Education Gap: Vouchers and Urban Schools (Washington, D.C.: Brookings 
Institution Press, 2002), pp. 49-52. 

1 “Weighting is a procedure used to compensate for participant attrition. Using this 
procedure, some participants who returned for follow-up data collection represent both 
themselves and others who are similar to them on characteristics measured at baseline but 
who did not return for follow-up data collection. 
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Table 3: Ethnicity of Mothers of Voucher Applicants Participating in Studies 


(In percentages) 




New York 


Dayton Washington 8 


African American, Non-Hispanic 


44 


70 


95 


White, Non-Hispanic 


5 


29 


1 


Hispanic 


47 


0 


3 


All other 


4 


1 


2 



a Total does not equal 100 percent due to rounding. 



Sources: Paul E. Peterson, Jay P. Greene, William G. Howell, and William McCready, Initial Findings 
from an Evaluation of School Choice Programs in Washington, D.C., and Dayton, Ohio (Paper 
presented at the annual meeting of the Association of Public Policy and Management. New York City, 
October 1998).; Paul E. Peterson, David Myers, Josh Haimson, and William G. Howell, Initial Findings 
from the Evaluation of the New York School Choice Scholarships Program (Washington D C • 
Mathematica Policy Research, November 1997). 



Of the three studies, the New York study showed the greatest positive 
effect on voucher users’ academic achievement. In each of the 3 years of 
the New York study, African American students who used a voucher to 
attend private school performed better on the reading exams than a 
comparable control group. African American voucher users also performed 
better in math than those who did not receive a voucher and remained in 
public schools. 11 Tliis improvement in math scores was found in years 
1 and 3, but not year 2, of the study. 12 Achievement gains, however, did not 
extend to Hispanic voucher users. The New York study found that Hispanic 
students who used privately funded vouchers to switch from public to 



"Professor Alan Krueger and Pei Zhu of Princeton University have analyzed the data 
pertaining to the New York experiment. They explore some of the assumptions underlying 
the original research, such as the exclusion of students with missing baseline scores (who 
increase the original sample by over 40 percent) and the definition of race. Their findings 
raise doubts about the size and significance of earlier findings of a positive effect of 
vouchers on test scores for African American students. {Another Look at the New York City 
School Voucher Experiment, Alan Krueger and Pei Zhu, August 16, 2002, paper presented at 
the Conference on Randomized Experimentation in the Social Sciences at Yale University). 

According to results from the final year of the New York study, on average, African 
American students who used a voucher to attend private school in at least one year of the 
study ranked 9.7 percentile points higher in math and 5.4 percentage points higher in 
reading than the control group. (Average math percentiles were 25.5 for voucher users and 

15.8 for the control group. Average reading percentiles were 25.3 for voucher users and 

19.9 for the control group.) 



African American Voucher 
Users in New York City Had 
Improved Test Scores 
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private school had math and reading scores that were statistically the same 
as those of the control group. The New York study did not have sufficient 
numbers of white students or students from other ethnic groups to 
estimate program effects for them. In addition, when the New York study 
sample is considered as a whole — pooling together African Americans, 
Hispanics, and white students— there is no significant difference in 
achievement gains between voucher users and nonusers. Although the 
researchers conducted additional analyses to explain why these positive 
effects were seen for African American students and not for others, the 
cause of this difference remains unclear. In Dayton, African American 
voucher users showed improvements in reading in the second year of the 
study, an effect that approached, but did not reach, statistical 
significance. 13 No other effects on test scores were found for African 
American students or for students of any other racial or ethnic background. 

In the Washington, D.C., study, the academic achievement of African 
American students who used a voucher to switch from public to private 
school was not consistently higher over the 3 years of the study than that of 
the control group of African American students who remained in public 
schools. In the first year of the study, African American voucher users 
scored better than control group students in math, but worse in reading. In 
the second year, African American voucher users scored significantly 
better in both math and reading than the students who remained in public 
schools. In the third and final year of the study, however, there was no 
difference between the combined math and reading test scores of African 
American voucher users and those of the control group. Because a large 
majority of participants in the Washington study were African American, 
there were not enough students of other ethnic groups to make a reliable 
estimate of program effects. 

Several limitations prevent reaching more definitive conclusions from this 
body of research. Substantial numbers of both voucher recipients and 
nonrecipients who were tested at baseline did not return for follow-up 



13 This result reached the 90 percent confidence level, an effect that the researchers consider 
statistically significant. We typically use the 95 percent confidence level for determining an 
effect to be statistically significant. However, we recognize that with smaller sample sizes as 
in the Dayton study statistical significance at the 95 percent level is more difficult to 
achieve. 

‘“^he studies did not provide estimates of the separate math and reading scores for the third 
year of the Washington study. 
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testing in subsequent years (program attrition). 15 The researchers 
compensated for this by weighting the results of those who did return, but 
in Dayton and Washington, the problem of attrition from the studies was 
great enough that it was unclear if the procedures used could address it 
sufficiently. In addition to the problem of attrition from the study, the 
smaller initial sample size in Dayton may have resulted in estimates that 
were less precise and not statistically significant. In the Washington, D.C., 
study, another factor limited definitive study conclusions. Specifically, only 
68 percent of those who were offered a voucher and who returned for 
follow-up testing the first year had actually used the voucher to attend a 
private school. This number declined to 47 percent and 29 percent in the 
second and third years, respectively. The analytical procedures used in the 
studies are more effective when a higher percentage of students who were 
offered vouchers actually use them to attend a private school. The New 
York study had the fewest problems with attrition and voucher use, and the 
estimated effects from the New York study are probably the most reliable. 
Finally, to our knowledge, at the time this report was written, only the New 
York data had been examined by researchers who were not part of the 
original research team. Confidence in the conclusions drawn from these 
studies will be enhanced when other researchers reanalyze these data and 
examine the assumptions underlying the original research. 

The findings of the studies show positive achievement gains 16 for low- 
income African American voucher users in New York City who applied for 
vouchers while in grades 1-4; however, the findings do not provide evidence 
that African American students elsewhere, or any other students, would 



“In New York, the rate of students returning for follow-up testing in subsequent years of the 
study ranged from 82 percent to 66 percent. In Dayton, this rate ranged from 56 percent to 
49 percent, and in Washington, it ranged from 63 percent to 50 percent. According to the 
researchers, approximately equal percentages of voucher recipients and nonrecipients 
returned for follow-up testing. The researchers used incentives to maximize the number of 
nonrecipient students returning. 

The research team indicates that the positive effect on African American students of using 
vouchers could diminish the achievement gap between them and white students by roughly 
one-third. However, Alan Krueger, of Princeton University, disputes the estimate offered by 
the research team and the standard deviation used in deriving it. He estimates that, when 
the appropriate standard deviation is applied, the impact of attending private school using a 
voucher would diminish the achievement gap by one-fifth to one-quarter. Furthermore, he 
argues that from a policy perspective, it is more relevant to analyze the impact of a voucher 
offer, rather than looking only at those who chose to use it. Krueger estimates that the 
impact of offering vouchers would be to diminish the achievement gap by no more than one- 

eighth. (Education Next, winter 2001, pp. 4-5). 
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realize achievement gains if they were offered vouchers to attend private 
schools. Differences in how voucher programs are designed and 
implemented, as well as differences in the participating students and in the 
local public and private schools affected by such programs, make it 
impossible to predict the effects of larger scale programs or programs 
similar in scale in different cities. The authors of the studies caution against 
generalizing from these results to a larger scale program involving all 
children in a large urban school system. They point out that only a small 
percentage of low-income students in the three cities received vouchers, 
and that the outcomes of a larger program could be quite different if the 
applicants to the study programs differ from eligible public school students 
in general . 17 Additionally, the low-income families in the studies may 
represent an even more distinct group: low-income as defined by program 
eligibility requirements, but able to pay at least the minimum amount of the 
private school tuition required by the program. 



Parents Consistently 
Reported Satisfaction with 
Private School Instructional 
Programs, Teachers, and 
Environment 



In addition to measuring student academic achievement, the studies of 
privately funded voucher programs in New York, Dayton, and Washington, 
D.C., used surveys to measure parents’ perceptions of their children’s 
schools. The research team found that parents who used a voucher to send 
their children to private schools were more likely to be satisfied with their 
children’s education overall and perceived their children’s schools to be 
better on a number of indicators, compared to parents in the control 
groups. These findings held true for all parents of voucher users, not only 
for African Americans . 18 In all three cities in each year for which data are 
available, parents of voucher users were more likely than parents of 
control group students to give their child’s school an “A” on an A to F 
scale . 19 In all three cities, in at least one study year, when asked about 
specific aspects of their children’s schools, the parents of voucher users 



17 Howell and Peterson, The Education Gap , 166-167. 

18 Researchers used analytical techniques to address attrition (students no longer 
participating in the evaluation) and address that some students offered vouchers did not use 
them and others not offered vouchers attended private school without them. For additional 
information on these techniques, see Howell and Peterson, The Education Gap, pp. 43-47; 
49-52. 



19 In the second year of the Washington study, the study reported that the parents of voucher 
users were more likely to give a grade of “A or B”, not “A.” Although the Washington study 
lasted for 3 years, no third year survey results had been released at the time this report was 
written. 
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were more likely than the parents of control group students to say they 
were “very satisfied” with school safety, teaching, and school curricula. In 
all three years in New York, and in the second year in Dayton, the parents 
of voucher users were more likely to report being very satisfied with the 
academic quality of their child’s school than were the parents of students 
who did not receive a voucher. 20 Parents of voucher users were also more 
likely to be very satisfied with discipline in their child’s school than were 
the parents of control group students in all 3 years of the New York study 
and in the first year of the Dayton and Washington studies. 

According to parents surveyed, the private schools attended by voucher 
users in New York; Dayton; and Washington, D.C., had significantly 
different characteristics than the public schools attended by the control 
group students. For example, in all three studies in every year for which 
data were available, 21 the parents of voucher users reported their children’s 
schools had fewer students than did parents of the control group students. 
In at least 1 year for each of the three cities, parents of voucher users also 
reported that their children’s schools had smaller classes, were more likely 
to offer individual tutoring, and communicated with parents more 
frequently than did parents whose children remained in public schools. 
However, all three studies found that parents of voucher users reported 
that their children’s schools were less likely to have certain facilities. For 
example, based on parents’ reports, all three studies in at least 1 year found 
that the private schools used by voucher families were less likely than 
public schools attended by the control group students to have a nurse’s 
office. In New York and Washington, parents of voucher users were also 
less likely to report that their children’s schools had a cafeteria or offered 
programs for students with learning problems and non-English speakers 
than were parents of the control group students. 22 



20 In the first year follow-up surveys for the Dayton and Washington studies, researchers 
asked parents about their children’s academic program rather than academic quality and 
found that the parents of voucher users were more likely than the control group parents to 
be very satisfied. During second year follow-up in Washington, the parents of voucher users 
were no more likely to report being very satisfied with academic quality than those in the 
control group. 

21 As indicated in footnote 16, although the Washington study lasted for 3 years, no third year 
survey results had been released at the time this report was written. 

22 In Dayton, parents reported that public schools were no more likely to have a library or 
cafeteria or to offer special education programs than private schools, but participants were 
not asked about programs for non-English speakers. 

24 
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The studies of privately funded voucher programs in New York; Dayton; 
and Washington, D.C., found that the parents of voucher users reported less 
disruption in their children’s schools compared to the parents of control 
group students. Parents of voucher users were less likely than parents of 
control group students to report that fighting, truancy, cheating, or 
destruction of property were serious problems in their children’s schools. 
These results were true in all three studies in each year for which results 
were available, except for the second year of the Washington study, when 
only fighting showed a statistically significant difference. 

As in their analysis of test scores, several issues prevented the studies from 
resulting in more precise estimates of the effects of the voucher programs 
on parental perceptions of their children’s education. In Dayton and 
Washington, many results that were statistically significant in the first year 
of the studies were no longer significant in the second year. This may have 
been due to the low rates of study participants that returned for follow-up 
in both cities, the relatively small sample size in Dayton, and the low rate of 
voucher use and private school attendance by those offered vouchers in 
Washington. Despite these limitations, many of the effects were strong 
enough that statistically significant differences were found between the 
parents of voucher users and the parents of control group students. As with 
the analysis of test scores, the New York study had the least problems with 
participants not returning for follow up and families declining voucher 
offers, and probably produced the most reliable estimates. 

These three studies constitute an important first step toward 
understanding the effect of school vouchers on certain low-income (and in 
the case of test score effects, African American) students. However, their 
results cannot be taken as evidence of the effects of vouchers on other 
types of students in any other settings. Furthermore, to our knowledge, at 
the time this report was written, only the New York data had been 
examined by anyone other than the original research team. Further 
analyses of these data by other researchers and additional high quality 
studies in other settings and involving different types of students and 
schools are important next steps in informing this ongoing public debate. 


Agency Comments 


We received comments on a draft of this report from the Department of 
Education. These comments are in appendix II. Education generally 
agreed with the report and provided technical comments that we 
incorporated where appropriate. Education also agreed that more 
research is needed on the effects of expanded parental choice. We also 
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received technical comments from the researchers at Harvard and 
Mathematica whose work we assessed and from officials at the Children’s 
Scholarship Fund and Children First America. These comments were also 
incorporated where appropriate. 



As agreed with your office, unless you publicly announce the contents of 
this report earlier, we plan no further distribution until 16 days from the 
report date. At that time, we will send copies of this report to the Secretary 
of Education, appropriate congressional committees, and other interested 
parties. In addition, the report will be available at no charge on the GAO 
Web site at http://www.gao.gov. 

If you or your staff have any questions or wish to discuss this material 
further, please call me at (202) 512-7215 or David Beilis at (415) 904-2272. 

Sincerely yours, 




Mamie S. Shaul 

Director, Education, Workforce, 
and Income Security Issues 
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Appendix I 



Description of the Privately Funded Voucher 
Studies Reviewed 



The studies of privately funded voucher programs in New York; Dayton, 
Ohio; and Washington, D.C., were similar and rigorous in both design and 
implementation. All three were longitudinal studies, tracking participants 
over a 2 or 3 academic year period. In all three cities, researchers awarded 
vouchers randomly by lottery, thus maximizing the chances that those who 
received a voucher and those who did not receive a voucher (control 
group) were equivalent on those characteristics that could affect test score 
outcomes. At the outset of each study, researchers collected baseline 
data — information about the participants before the voucher program 
began — to help ensure equivalence of study groups and to compare against 
later outcomes. In addition to basic demographic information, the baseline 
data collected by the researchers included student achievement and 
parental satisfaction data. The researchers administered the math and 
reading sections of the Iowa Test of Basic Skills to applicant students who 
would be entering grades two and higher in the coming fall, when voucher 
use began. While the students were taking the tests, the accompanying 
parents and guardians filled out a survey that included questions about 
their satisfaction with a variety of aspects of their children’s education. The 
researchers required applicants to participate in the baseline tests and 
surveys as a condition of entering the random lottery to win a privately 
funded voucher. 

In each subsequent year of the studies, researchers again administered 
math and reading tests to study participants and asked their parents to fill 
out surveys. The researchers invited both voucher and control group 
members back for this follow-up data collection. To ensure that as many 
participants as possible returned for follow-up, the researchers offered 
incentives for the control group members and told voucher group members 
that their participation was required as a condition of continuing to receive 
a voucher. 23 Each year, the researchers analyzed the test scores to look for 
achievement differences between voucher and nonvoucher students and 
analyzed the survey results for differences in parental satisfaction levels 
between the two groups. 

For all three studies, the researchers took two steps to ensure the validity 
of their results. First they weighted their data to adjust for participants who 
failed to return for follow-up data collection. The researchers used the 
known characteristics of study participants (e.g., family size, mother’s 
education, race, etc.) to estimate the probability that any given individual 



23 No one’s voucher was actually terminated because of failure to participate in follow-up. 
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Appendix I 

Description of the Privately Funded Voucher 
Studies Reviewed 



would return for follow-up testing. Using these calculated probabilities, 
they modified their data so that the responses of those who returned for 
follow-up testing, who were similar on these characteristics to the 
nonretumers, were counted more heavily in the analysis. This means that 
participants who did not return were represented by returning participants 
who were similar to them. 24 

The second way the researchers improved the quality of their results was 
by using a statistical technique called an instrumental variable analysis to 
compensate for the fact that not all students who were offered a voucher 
actually used it to attend private school, and some control group students 
who were not offered a voucher attended private school anyway. Because 
of this crossover between the voucher offer and control groups, a 
comparison of outcomes between those offered a voucher and those not 
offered a voucher gave only the effect of the offer itself, not the effect of 
actually using a voucher to attend a private school. The instrumental 
variable analysis involved a statistical procedure in which the researchers 
used a student’s status as a voucher recipient or nonrecipient to predict 
whether that student attended private school. The researchers then used 
these results in a second statistical model, which gave the effect of actually 
using a voucher to attend a private school on test scores and survey 
results. 25 

Tables 4, 5, and 6 summarize the results and key strengths and weaknesses 
of the New York City; Dayton; and Washington, D.C., studies. They also 
provide our interpretation of the degree to which the studies provide 
evidence of any effect that voucher use had on test scores and parental 
satisfaction. 



24 For additional detail on the weighting procedure used in the studies, see Howell and 
Peterson, The Education Gap, pp. 209-216. 

/ 

25 For further details on the instrumental variable technique and other aspects of the studies’ 
design and data analysis methods, see Howell and Peterson, The Education Gap , pp. 43-47- 
49-52. 
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Description of the Privately Funded Voucher 
Studies Reviewed 




Table 4: New York City Study Summary— 1997 through 2000 (3 academic years) 



Study documents 

Daniel Mayer, Paul Peterson, David Myers, Christina 
Clark Tuttle, and William Howell, School Choice in 
New York City after Three Years: An Evaluation of 
the School Choice Scholarships Program 
(Washington, D.C.: Mathematica Policy Research, 
February 2002). 

William Howel! and Paul Peterson, The Education 
Gap: Vouchers and Urban Schools , (Washington, 
D.C.: The Brookings Institution, 2002). 

William Howell, Patrick Wolf, Paul Peterson and 
David Campbell, Test-Score Effects of School 
Vouchers in Dayton, Ohio , New York City, and 
Washington, D.C.: Evidence from Randomized Field 
Trials (Paper presented at the annual meeting of the 
American Political Science Association, Washington, 
D.C., Sept. 2000). 



Reported results 

• Study reports positive and statistically significant 8 differences on reading, math, 
and composite test scores between African American Voucher users and the 
control (no voucher) group. bc These results held for all 3 years of the study, 
except for math score differences which were not significant in the second year. 

• Study reports no significant differences between test scores of non-African 
American voucher and control group students.. The non-African American 
students in this study were primarily of Hispanic ethnicity. 

• On ai range of issues on which parents were surveyed, the voucher group had 
the following outcomes in all 3 years of the study when compared to the control 
(no voucher) group: 

-Higher parental satisfaction (percent of parents giving the school a report card 
grade of “A,” and reporting satisfaction with academic quality, quality of 
teaching, curriculum, and school safety and discipline). 

—Attended smaller class and smaller schools. 

-More access to computer labs and tutors. 

-Less disorderly behavior (fighting, cheating, truancy, destruction of property). 
-More homework. 

-More communication from school to parents. 

-Less access to a cafeteria, a nurse’s office, and services for non-English 
speakers and those with learning problems. 
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Appendix I 

Description of the Privately Funded Voucher 
Studies Reviewed 



Key strengths 



Key limitations 



Evidence of impact 



• The study design involved vouchers being 
offered at random to eligible families. This 
helps ensure the equivalence of the 
characteristics of students using vouchers 
to attend private school and those of the 
control group. 

• An analytical technique — the instrumental 
variable approach* 1 — was used to adjust 
for factors that threatened the equivalence 
of the study groups (e.g., some students 
offered vouchers did not use them to 
attend private schools and other students 
not offered vouchers attended private 
school without them). 

• The researchers used an analytical 
technique — weighting 6 — to adjust for 
observed differences between the 
students who returned for testing and 
surveys 1, 2, and 3 years after the 
voucher program began and those who 
did not remain in the study. 



♦ The researchers acknowledge that these 
results cannot be generalized to larger scale 
programs, programs in other sites, programs 
serving other than low-income families or, in 
the case of the test score improvements, to 
groups other than African American 
students who applied for vouchers while in 
grades 1-4. 

• The analysis did not adjust for the possibility 
of numerous students attending the same 
school. Thus, it did not completely rule out 
the possibility that the positive effects were 
actually due to attendance at specific 
schools, rather than voucher use itself. 



• Evidence for positive impact on test 
scores for African American students. 

• No evidence of test score effects for 
Hispanic students. 

• Strong evidence for parental survey 
results as reported in “results” column, 
including increased parental satisfaction 
by voucher parents. 



a We typically use the 95 percent confidence level for determining an effect to be statistically significant. 

Composite test scores are the average of the math and reading scores. 

c For ease of discussion, we use the terms voucher users and control (no voucher) group. The 
comparison of these groups involves students who used vouchers to attend private school and those 
who were likely to have used vouchers had vouchers been offered to them. 

^The instrumental variable approach involves a statistical procedure in which the researchers used a 
student’s status as a voucher recipient or non-recipient in a statistical model to predict whether that 
student attended private school. The researchers then use these results in a second statistical model, 
which gives the effect of using a voucher on test scores and survey results. This final model can be 
thought of as comparing those who attended private school with a voucher to those who did not but 
who would have attended private school had a voucher been offered to them. 

^Weighting is a procedure used to compensate for participant attrition. Using this procedure, some 
participants who returned for follow-up data collection represent both themselves and others who are 
similar to them on characteristics measured at baseline but who did not return for follow-up data 
collection. 
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Description of the Privately Funded Voucher 
Studies Reviewed 




Table 5: Dayton, Ohio, Study Summary — 1998 through 2000 (2 academic years) 



Study documents 



Reported results 



Martin West, Paul Peterson, and David Campbell, School 
Choice in Dayton , Ohio, After Two Years: An Evaluation of 
the Parents Advancing Choice in Education Scholarship 
Program (Cambridge, Mass.: Program on Education 
Policy and Governance, Harvard University, Mass.: 
August 2001). 



• Study reports positive effects that approach, but do not reach, statistical 
significance on reading and composite scores for African American voucher 
users in the second year of the study. a,b,f There were no other positive 
effects on reading, math, or composite test scores for voucher users of any 
other racial or ethnic background. The study participants were primarily 
African American and Caucasian. 



William Howell and Paul Peterson, School Choice in 
Dayton, Ohio: An Evaluation after One Year (Paper 
presented at the Conference on Vouchers, Charters and 
Public Education sponsored by the Program on Education 
Policy and Governance, Harvard University, Cambridge, 
Mass.: March 2000). 

William Howell and Paul Peterson, The Education Gap: 
Vouchers and Urban Schools, (Washington, D.C.: The 
Brookings Institution, 2002). 

William Howell, Patrick Wolf, Paul Peterson and David 
Campbell, Test-Score Effects of School Vouchers in 
Dayton, Ohio, New York City, and Washington, DC. : 
Evidence from Randomized Field Trials (Paper presented 
at the annual meeting of the American Political Science 
Association, Washington, D.C., Sept. 2000). 



• On a range of issues on which parents were surveyed, the voucher group 
had the following outcomes in both years of the study when compared to 
the control (no voucher) group: 0 

-Higher parental satisfaction (percent of parents giving the school a report 
card grade of “A,” and reporting satisfaction with quality of teaching, and 
curriculum). 

-Attended smaller schools. 

-Less disorderly behavior (fighting, cheating, truancy, destruction of 
property). 

-More access to music programs. 
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Description of the Privately Funded Voucher 
Studies Reviewed 




Key strengths 

• The study design involved vouchers 
being offered at random to eligible 
families. This helps ensure the 
equivalence of the characteristics of 
students using vouchers to attend 
private school and those of the 
control group. 

• An analytical technique — the 
instrumental variable approach— was 
used to adjust for factors that 
threatened the equivalence of the 
study groups (e.g. , some students 
offered vouchers did not use them to 
attend private schools and other 
students not offered vouchers 
attended private school without 
them). 

• The researchers used an analytical 
technique — weighting 6 — to adjust for 
observed differences between the 
students who returned fortesting and 
surveys 1 and 2 years after the 
voucher program began and those 
who did not remain in the study. 



Key limitations 

• The researchers acknowledge that these results 
cannot be generalized to larger scale programs, 
programs in other sites, or programs serving other 
than low-income families who applied for vouchers 
while in grades 1-8. 

• High levels of attrition (students not continuing to 
participate in data collection for the study) and low 
rates of voucher use (only 60 percent of those 
offered a voucher used it for both years) may make 
it difficult to reliably estimate effects. 



Evidence of impact 

• No evidence of test score effects for 
students of any racial/ethnic 
background. 

• Strong evidence for parental survey 
results as reported in “results” column, 
including increased parental 
satisfaction by voucher parents. 



• Researchers did not account for the unequal 
probability of selection of children within families 
(selection was based on family units but 
observations were on children). 



• The analysis did not adjust for the possibility of 
numerous students attending the same school. 
Thus, it did not completely rule out the possibility 
that the positive effects were actually due to 
attendance at specific schools, rather than voucher 
use itself. 



a We typically use the 95 percent confidence level for determining an effect to be statistically significant. 

Composite test scores are the average of the math and reading scores. 

c For ease of discussion, we use the terms voucher users and control (no voucher) group. The 
comparison of these groups involves students who used vouchers to attend private school and those 
who were likely to have used vouchers had vouchers been offered to them. 

"The instrumental variable approach involves a statistical procedure in which the researchers used a 
student’s status as a voucher recipient or non-recipient in a statistical model to predict whether that 
student attended private school. The researchers then use these results in a second statistical model, 
which gives the effect of using a voucher on test scores and survey results. This final model can be 
thought of as comparing those who attended private school with a voucher to those who did not but 
who would have attended private school had a voucher been offered to them. 

"Weighting is a procedure used to compensate for participant attrition. Using this procedure, some 
participants who returned for follow-up data collection represent both themselves and others who are 
similar to them on characteristics measured at baseline but who did not return for follow-up data 
collection. 

’Reading and composite score effects for African American students in Dayton in the second year of 
the study reached a 90 percent confidence level, an effect that the researchers consider statistically 
significant. Although we typically use the 95 percent confidence level for determining an effect to be 
statistically significant, we recognize that with smaller sample sizes, as in the Dayton study, statistical 
significance is more difficult to achieve. 
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Description of the Privately Funded Voucher 
Studies Reviewed 




Table 6: Washington, D.C., Study Summary — 1998 through 2001 (3 academic years) 



Study document 

Patrick Wolf, Paul Peterson, and Martin West, 
Results of a School Voucher Experiment: The 
Case of Washington, D.C., after Two Years 
(Paper presented at the annual meeting of the 
American Political Science Association, San 
Francisco, Calif., August 2001). 

Patrick Wolf, William Howell, and Paul Peterson, 
School Choice in Washington, D.C.: An 
Evaluation after One Year (Paper presented at 
the Conference on Vouchers, Charters, and 
Public Education sponsored by the Program on 
Education Policy and Governance, Harvard 
University, Cambridge, Mass., March 2000). 

William Howell and Paul Peterson, The 
Education Gap: Vouchers and Urban Schools, 
(Washington, D.C.: The Brookings Institution, 
2002 ). 

William Howell, Patrick Wolf, Paul Peterson and 
David Campbell, Test-Score Effects of School 
Vouchers in Dayton, Ohio, New York City, and 
Washington, D.C.: Evidence from Randomized 
Field Trials (Paper presented at the annual 
meeting of the American Political Science 
Association, Washington, D.C., Sept. 2000). 



Reported results 

• Year 1 of the 3-year study reports statistically significant, 8 positive effects on math 
scores but statistically significant, negative effects on reading scores for African 
American voucher users as opposed to the control (no voucher) group. 6 During the 
second year, there were significant, positive effects on reading, math, and composite 
test scores 0 for African American voucher users. In the third and final year of the 
study, significant effects on composite scores disappear, and no information is 
currently available regarding the third year results for math and reading individually. 

• Study reports no differences between test scores of non-African American voucher 
and control group students. However, only a small percentage of the students in the 
study were not African American. 

• On a range of issues on which parents were surveyed, the voucher group had the 
following outcomes in both years of the study for which data are available' when 
compared to the control (no voucher) group: 

--Higher parental satisfaction (percent of parents giving the school a report card 
grade of “A”, and reporting satisfaction with curriculum and school safety). 
—Attended smaller class and smaller schools. 

-Less fighting. 

-More communication from school to parents. 

-Less access to computer labs, a cafeteria, a nurse’s office, special education 
services, and services for non-English speakers and those with learning problems. 
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Description of the Privately Funded Voucher 
Studies Reviewed 



Key strengths 

• The study design involved vouchers 
being offered at random to eligible 
families. This helps ensure the 
equivalence of the characteristics of 
students using vouchers to attend 
private school and those of the control 
group. 

• An analytical technique— the 
instrumental variable approach— was 
used to adjust for factors that threatened 
the equivalence of the study groups 
(e.g., some students offered vouchers 
did not use them to attend private 
schools and other students not offered 
vouchers attended private school 
without them). 

• The researchers used an analytical 
technique — weighting 6 — to adjust for 
observed differences between the 
students who returned for testing and 
surveys 1 , 2, and 3 years after the 
voucher program began and those who 
did not remain in the study. 



Key limitations 

• The researchers acknowledge that these results cannot be 
generalized to larger scale programs, programs in other 
sites, programs serving other than low-income families or, 
in the case of the test score improvements to groups other 
than African American students who applied for vouchers 
while in grades 1 -8. 

• High levels of attrition (students not continuing to 
participate in data collection for the study) and low rates of 
voucher use (only 29 percent of those offered a voucher 
used it for all 3 years) may make it difficult to reliably 
estimate effects. 

• Researchers did not account for the unequal probability of 
selection of children within families (selection was based on 
family units but observations were on children). 



Evidence of impact 

• While there is evidence for 
positive impact on overall 
test scores for African 
American students in the 
second year of the study, 
these positive impacts 
disappear in the 3rd and final 
year. 

• Strong evidence for parental 
survey results as reported in 
“results” column, including 
increased parental 
satisfaction by voucher 
parents. 



• The analysis did not adjust for the possibility of numerous 
students attending the same school. Thus, it did not 
completely rule out the possibility that the positive effects 
were actually due to attendance at specific schools, rather 
than voucher use itself. 



a We typically use the 95 percent confidence level for determining an effect to be statistically significant. 

b For ease of discussion, we use the terms voucher users and control (no voucher) group. The 
comparison of these groups involves students who used vouchers to attend private school and those 
who were likely to have used vouchers had vouchers been offered to them. 

Composite test scores are the average of the math and reading scores. 

^he instrumental variable approach involves a statistical procedure in which the researchers used a 
student’s status as a voucher recipient or non-recipient in a statistical model to predict whether that 
student attended private school. The researchers then use these results in a second statistical model, 
which gives the effect of using a voucher on test scores and survey results. This final model can be 
thought of as comparing those who attended private school with a voucher to those who did not but 
who would have attended private school had a voucher been offered to them. 

Weighting is a procedure used to compensate for participant attrition. Using this procedure, some 
participants who returned for follow-up data collection represent both themselves and others who are 
similar to them on characteristics measured at baseline but who did not return for follow-up data 
collection. 

‘Survey results were not available for the third year of the Washington, D.C. study. 
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Comments from the Department of Education 



THE SECRETARY OF EDUCATION 

WASHINGTON, D.C. 20202 

August 9, 2002 



Ms. Mamie S. Shaul 
Director, Education, Workforce, 
and income Security Issues 
U.S. General Accounting Office 
Washington, D.C. 20548 

Dear Ms. Shaul: 

Thank you for the opportunity to comment on the recent GAO report, School Vouchers : 
Characteristics of Privately Funded Programs. As I am sure you are aware, expanding 
parental options and information is one of the four principles of President Bush’s 
education agenda, along with ensuring accountability for results, strengthening local 
control and flexibility, and basing programs on sound, scientific research. As your report 
indicates, the parental choice movement has grown tremendously in just the last decade, 
thanks in large part to the same private scholarship programs you have examined. 

Your report also reveals the varied nature of private scholarship programs, which operate 
in some 38 States nationwide and the District of Columbia. Smaller programs offer just a 
handful of scholarships, while larger organizations, like the local Children’s Scholarship 
Fund program in New York City, award up to 3,100 scholarships annually. Scholarship 
amounts also vary, ranging from just a few hundred dollars to well over $1,500 annually 
in other programs. Because of the success of these innovative programs, Arizona, 
Florida, and Pennsylvania have all passed tax credit laws, encouraging charitable 
contributions to private scholarship organizations, and Utah and other States are 
contemplating similar legislation. While these types of tax credits represent one 
approach to expanding choice for disadvantaged families, States are exploring a variety 
of different methods, including tax credits for low-income families to send their own 
children to schools of choice and directly funding school choice experiments, like the 
well-known programs in Cleveland, Milwaukee, and Florida. 

Your report indicates that initial findings from private scholarship programs in Dayton, 
Ohio, New York City, and Washington, D.C., suggest that expanded parental choice can 
boost academic achievement for African-American students. However, you also caution 
these findings are limited in scope given the small size of the evaluation. In order to 
develop more information on the effects of expanded parental choice, we need more 
research and experimentation. 

That is why President Bush has requested $50 million in his FY 2003 Department of 
Education budget for a Choice Demonstration Fund. This fund would support research 



Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation. 
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Page 2 - Ms. Mamie Shaul 



projects that develop, implement, : and evaluate innovative approaches to providing 
parents with expanded options, including both public and private school choice. 
President Bush has also proposed a federal tax credit for low-income families whose 
children are trapped in low-performing schools to help pay for tuition and other expenses 
at a local private school. It is also why I applaud Representative Dick Armey for his 
efforts to expand school choice for disadvantaged families here in the District of 



Support for expanded parental options and information is found elsewhere in the 
President’s budget. We have proposed $200 million to help fund the development of 
charter schools, $100 million to help charter schools address urgent facilities needs and 
$25 million for voluntary public school choice programs. These initiatives are part of our 
broader efforts to improve public education. As a longtime educator and former 
superintendent, 1 understand that greater parental choice and competition are not threats 
to public education but pieces of their salvation. We should not shy away from 
competition and innovation but embrace it. 

This is an exciting time for the parental choice movement. The recent U.S. Supreme 
Court ruling upholding the Cleveland school choice plan validates that such programs are 
Constitutional and will hopefully encourage States to experiment more widely with this 
promising reform. But we would have never gotten to this point had it not been tor the 
pioneering work of the private scholarship programs you identify in your report. I 
applaud groups such as the Children’s Scholarship Fund, Children First America , and the 
many local organizations that have made it their mission to ensure that all children are 
educated and no child is left behind. 1 trust these groups, and Federal, State, and local 
governments, will do all they can to expand choices and options to thousands more 
children in the years to come. 



Columbia. 



Sincerely, 
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